In inductive transfer learning, fine-tuning pre-trained convolutional
networks substantially outperforms training from scratch. When using
fine-tuning, the underlying assumption is that the pre-trained model extracts
generic features, which are at least partially relevant for solving the target
task, but would be difficult to extract from the limited amount of data
available on the target task. However, besides the initialization with the
pre-trained model and the early stopping, there is no mechanism in fine-tuning
for retaining the features learned on the source task. In this paper, we
investigate several regularization schemes that explicitly promote the
similarity of the final solution with the initial model. We show the benefit of
having an explicit inductive bias towards the initial model, and we eventually
recommend a simple $L^2$ penalty with the pre-trained model being a reference
as the baseline of penalty for transfer learning tasks.

本文研究在归纳迁移学习中，fine-tuning 方式相较于从头训练卷积神经网络具有更好的表现。为了提高 fine-tuned 模型在目标任务上的泛化性能，本文探究了多种正则化方法，最终推荐采用 $L^2$ 惩罚并将预训练模型作为基准的方法作为迁移学习任务的 baseline。